Analyzing Disagreements
نویسندگان
چکیده
We address the problem of distinguishing between two sources of disagreement in annotations: genuine subjectivity and slip of attention. The latter is especially likely when the classification task has a default class, as in tasks where annotators need to find instances of the phenomenon of interest, such as in a metaphor detection task discussed here. We apply and extend a data analysis technique proposed by Beigman Klebanov and Shamir (2006) to first distill reliably deliberate (non-chance) annotations and then to estimate the amount of attention slips vs genuine disagreement in the reliably deliberate annotations.
منابع مشابه
Some Basic Disagreements on Development
Once upon a time I made a list of ten policies which I thought more-or-less everyone in Washington could agree were needed more-or-less everywhere in Latin America. I found a common interpretation of what I had written to be quite extraordinary, indeed largely the contrary of what I was arguing, so rather than try once again to stress the things on which we agree, let me try analyzing the disag...
متن کاملThe Utah paradigm on animal models of skeletal disorders: quo vadis?
Skeletal disorders that need effective studies in suitable animal models include "osteoporosis", arthroses and hard and soft tissue healing. For people doing or analyzing such studies this article provides a brief overview and some salient implications of the Utah paradigm of skeletal physiology. The article leaves discussing and resolving any disagreements and controversies about such things t...
متن کاملInvestigating disagreements through a context-specific approach: A case of Iranian L2 speakers
The current study investigated the expression of disagreement by Iranian advanced English learners. The data for the study comprised the recorded discussions of 26 male and female interlocutors in three different settings: 1) language institute, 2) home environment, and 3) university setting. Analysis of the arguments pointed to the influence of c...
متن کاملDisagreements on Collegial Courts : A
How do disagreements between judges on collegial (multimember) courts affect legal policy? We address this question by developing an account of the nature of judicial disagreements in the case-space model of judicial choice. We distinguish between different types of disagreement, from disagreements about case facts to disagreements about how legal rules should treat varying case facts, and cons...
متن کاملIn the Eye of the Cyclops: The Classic Case of Cospeciation and Why Paradigms Are Important
Scientific disagreements due to empirical problems—not enough data, not enough of the critical type of data, problems in analyzing the data—are generally short-lived and resolved in the next cycle of data production. Such disagreements are thus transitory in nature. Persistent scientific conflicts, on the other hand, do not necessarily mean some facts are correct and some are wrong, nor do they...
متن کامل